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Description 

Cross-Reference to Related AppHcation 

X.^yTbis- appUcation claimG the bonofit of Baker et al/s -copending- 
United States Provisional Patent AppHcation No. 60/302,761, entitled 

10 "MOVING IMAGER CAMERA FOR TRACKING, SCANNING, 
RANGE AND SUPER-RESOLUTION," filed December 11, 1996, 
which is incorporated herein by reference in its entirety. This 
application is related to Woodfill et al.'s copending U.S. Patent 
AppHcation Serial No. 08/839,767, filed April 28, 1997, entitled 

15 "Data Processing System and Method," which is incorporated herein 
by reference in its entirety. 
Technical Field 

This invention relates generally to computer input devices, and 
more particularly to digital image capture devices used to provide 
20 ranging and tracking information for a computer system. 

Background Art 

The range of an object, Le. the distance to the object from an 
observation site, can be determined by the analysis of two or more 
spatially separated images (often referred to as "binocular images" 
25 when there are two images) that are taken from the observation site. 
In range computation from simultaneously acquired binocular digital 
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images, the area of processing is limited to the visible region of 
overlap between the two images. To maintain a reasonable region of 
overlap usually necessitates redirecting the optical axes of the 
cameras (ie. changing their vergence) which introduces other 
5 problems including a usual necessity to resample the imagery. As is 
well known to those skilled in the art, "vergence" means the angle 
between the optical axes of the lenses. 

The processing of binocular or multi-view imagery for range 
computation is easiest when the optical axes are parallel and the 
10 imaging surfaces are coplanar - in what is termed parallel epipolar 
^":3 geometry. Because verging the optical axes to optimize the region of 

H image overlap eliminates image-surface co-planarity, the complexities 

i n 

i.g of calculating range increases significantly with non-parallel viewing, 

m This is further compounded when viewing objects at a variety of 

15 azimuths and distances where adjustments in the view direction as 

□ well as verging would be necessary to retain sufficient image overlap. 

py For computational purposes, the frame of reference for scene 

description is usually tied to image location, so changing the image 
location through vergence adjustments necessitates reconfiguring the 
20 frame of reference. Again, adjusting a system's frame of reference 
increases the computational and conceptual complexity of its 
analysis. 

A similar situation arises for typical monocular (/.e. single 
image) computer analysis of tracking and scanning in some space 
25 before the camera. With subjects able to operate over a broad region 
before the camera, continued observation generally involves use of 
either wide-angle optics or a panning/tilting mechanism to properly 
direct the camera's view direction. These control mechanisms are 
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relatively complex, must move fairly large pieces of equipment 
(cameras and their lenses), and alter the underlying geometric frame 
of reference for scene analysis by rotating the frame of reference with 
the cameras. In addition, the use of wide angle optics works against 
5 high resolution analysis, as only larger scene detail is visible. 

One approach to solve these acquisition problems in image- 
based range and tracking computation would be to employ greatly 
oversized imagers (e.g. imagers having about 3K by 3K or 9x10^ 
elements), and select corresponding standard-sized windows within 
10 these for processing. However, such an approach would be 
■3 prohibitively expensive. For example, a IK by IK imager sells for 

^'4 well over a thousand dollars. Higher resolution imagers are available 

i.n at considerably greater price. 

i ?3 

A prior art solution to the apparent dichotomy between simple 
15 processing (with parallel epipolar geometry) and broad depth and 
tracking coverage exists in adaptation of perspective-correcting lens 
r|j systems as used in "view-camera" or "technical-camera" designs. In 

such designs, an oversized lens is used to image the scene, and lateral 
repositioning of the lens or imaging platform can be used to redirect 
20 the camera without rotating the imaging surface. For single camera 
use this enables maintaining lines parallel in the world parallel on the 
image plane; in ranging camera use it enables parallel epipolar 
geometry. 

For example, in U.S. Patents 5,063,441 and 5,142,357 of 
25 Lipton et al. , devices for use in 3D videography are disclosed. More 
particularly, Lipton et al. teach devices for capturing binocular 
images for use in stereo videography (stereo movies), with reduced 
viewing eyestrain, by using dual and triple camera systems. Briefly 


stated, Lipton et al teaches an imager controller for epipolar stereo 
capture in videography, including stereo lenses mounted fixedly 
together in a single housing. Stereographies Inc., of San Raphael, 
California, produces a product embodying elements of the Lipton et 
5 aL patents. 

In the matter of two-dimensional imager control, U.S. Patent 
5,049,988, of Sefton et al. teaches a system that provides the display 
of a video capture window for surveillance applications. Phillips , in 
U.S. Patent No. 4,740,839, teaches a TV surveillance system operated 
10 by sub-sampling a conventional camera, with a result that resembles 
the Lipton et al. approach of image capture. 

As will be appreciated, image capturing of the prior art uses 
planar sensors due to the high cost, lack of availability, and 
complexities involved with the use and manufacture of curved or 
15 "spherical" sensors. However, spherical sensors have a number of 
advantages with respect to field of view, view direction, and use in 
stereo image capture that designers of prior art digital imaging 
cameras have apparently failed to consider. 


Disclosure of the Invention 

The present invention includes a multi-image camera system 
for automated stereo ranging applications that provides high- 
resolution broad field coverage over a wide range of subject distances 
5 while enabling retention of the epipolar constraints necessary for 
efficient stereo matching. At the same time, the multi-image camera 
of the present invention supports monocular image applications such 
as object tracking. 

In one embodiment of the present invention, an imaging device 
10 is preferably placed on a three-degree-of-fireedom linear motion 
positioning mechanism whose position can be controlled by a 
computer. Broad field coverage is preferably attained through use of 
a wide-angle large-coverage lens (e.g. a lens designed for a 35mm 
camera). High resolution is attained through placing the imaging 
15 device under the large-coverage lens, so the imaging device's 

immediate field of view is considerably narrower than that provided 
by the lens. 

In contrast with traditional view-camera usage, the present 
invention teaches moving the imaging surface instead of the lens. 

20 This allows the apparatus to retain co-planarity of the images and, so 
long as the displacements can be determined to sub-pixel accuracy, 
maintains a stable frame of reference for the analysis, all while 
providing the required view re-directions. Moving the lens alters the 
projective relationships among observations, whereas moving the 

25 imager does not. 

Image focus may be attained through the traditional rotating- 
travel focus adjustment although, for quantitative computational 


tasks, this is often unsatisfactory as the center of projection may vary 
with lens rotation. For this situation, our preferred embodiment of the 
moving imager provides a back- focus capability, as will be described 
below. 

5 Computer control of the platform enables positioning accuracy 

in the micron range. Stepper motors, for example, operating with full 
step positioning, can position a platform along an axis to within a few 
microns of the desired position. Differential stepper control enables 
increasing the accuracy of this placement (the number of locations 
10 where this precision is attainable) by one to two orders of magnitude. 
Fine specification of absolute imager location can also be attained 
through use of interferometers or sonar-based ranging systems, as will 
be described below in more detail. 

j J The planar imaging embodiment of the present invention 

15 enjoys a number of advantages over related devices of the prior art. 
In the present invention, an oversized lens is used to provide a wide 
fy field of view of the scene. Movement of the imaging surface under 

\1 the lens provides high-resolution view selection, which is the 

equivalent of view redirection of pan and tilt motions without the 
20 need for a complex mechanism for accomplishing such motions. 

With accurate positioning knowledge, this form of imaging provides a 
stable frame of reference for scene computation. Accurate 
positioning information may be attained using the positioning and 
control systems described below. 

25 Furthermore, the apparatus of the present invention provides an 

economical solution to the aforementioned problems of the prior art. 
For example, the present invention is operable with relatively 
economical high-precision imagers that use displacement for 
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direction selection in tracking, scanning, and range computation. 
More particularly, certain embodiments of the present invention place 
an imaging device on a three-degree-of- freedom linear motion 
. platform whose position can be controlled by a computer to 
5 efficiently and economically provide the desired direction selection. 
The third degree of freedom provides back-plane focus control. 
Back-plane focusing has an advantage for traditional lens designs 
where focus from element rotational displacements causes image 
centering variations. 

10 In a spherical camera embodiment of the present invention, 

image capture is effected through spherical focal-plane imaging 
J using, for example, a spherical-faced fiber-optic faceplate. One lens 

,y suitable for this embodiment, the Baker Ball lens, is described in a 

wartime report referenced below. In this spherical-imaging 
15 embodiment of the present invention, a mechanism is provided that 
shares many of the design considerations of the above-described 
^ linear moving-imager camera. Preferably, this embodiment uses a 

|;3 high resolution imaging device (IK by IK elements with 12 bits of 

resolution at each pixel), mounted behind a fiber-optic faceplate that 
20 transfers the focal-plane light pattern to the sensor elements of the 
imager. Notable in this embodiment of the moving imager is that the 
lens system has no particular optical axis or axis along which any 
imaging surfaces are preferentially oriented. 

In addition, the spherical camera embodiment of the present 
25 invention has a number of advantages over prior art planar imaging 
cameras. For example, the spherical camera embodiment we describe 
has 1) excellent optical resolution attainable since the focal surface 
does not need to be made planar (the lens is diffraction limited); 2) 


greater simplicity due to the advantages of rotational over linear 
displacements; 3) a greatly increased undistorted field of view; 4) 
exponentially less lens illumination fall-off with eccentricity (cosine 
of radial displacement rather than the fourth power of the cosine); 5) 
5 greater effective pan and tilt velocities; and 6) opportunity to study 
(via simulation) aspects of human psychophysical behavior in a 
mechanism of similar geometry. 

Certain embodiments of the present invention, both of the 
linear and spherical lens, provide lenses in multiple housings to 
10 increase the flexibility, reliability, and range finding quality of the 
system. A monocular version of the present invention requires only a 
single lens, and multi-lens versions of the present invention can use 
two or more lenses for multi-image ranging analysis. 


15 


These and other advantages of the present invention will become 
apparent upon reading the following detailed descriptions and studying 
the various figures of the drawings. 


Brief Description of the Drawings 


Figure 1 is a diagrammatic view of a monocular imager in 
accordance with the present invention; 

5 Figure 2 is a diagrammatic view of a multi-camera (binocular 

in this case) imager in accordance with the present invention; 

Figure 3 is an illustration of an imager under a wide angle lens 
of the present invention; 

Figures 4 and 5 illustrate the use of a binocular imager in 
10 adjusting the range of available stereoscopic images in accordance 
with the present invention; 

Figures 6 and 7 illustrate an alignment mechanism in 
accordance with one embodiment of the present invention; 

Figures 8 and 9 provide diagrammatic views of a multi-camera 
15 imager wherein each moving imager utilizes a three platform 
positioning mechanism; and 

Figure 10 is an illustration of a spherical imager of the present 
invention. 


Best Modes for Carrying out the Invention 

With reference to Fig. 1, a monocular imager apparatus or 
moving imager "camera" 10 in accordance with one embodiment of 
the present invention will now be described. The moving imager 
5 camera 10 includes a lens 12, and an imaging array, imaging device, 
or "imager" 14. As indicated by the arrows 16, 17, 18, 19, 100, and 
102, the imager is capable of independent motion along the three 
degrees of freedom defined by an X-Y-Z axis. This means that the 
imager 14 can move both in-plane and out-of-plane with respect to 

10 the lens 12 for view selection and focus, and may also be 

synchronized for rectangular displacements 20 in the imager plane P 
to provide super-resolution frame integration. The displacements 20 
are preferably rectangular, although they are represented by a circle in 
this figure. As will be appreciated by those skilled in the art, 

15 "rectangular" means any appropriate movement of the imager in the 
plane P to provide the necessary local sampling. For example, these 
rectangular type displacements are described in Tagami et al.'s U.S. 
Patent No. 5,402,171, issued March 28, 1995. 

Preferably the motions of the moving imager camera 10 are 
20 controlled by a computer controller 21 through the use of suitable 
transducing mechanisms. While high precision encoders or the like 
may be workable, it is preferable that an extremely high precision 
measurement system such as one utilizing an interferometer or sonar 
based mechanism be used. For imparting motion, an actuator such as 
25 a stepper motor is suitable. A variety of commercially available 

actuators are suitable for imparting motion along the X, Y and Z axes. 
For example, a stepper motor, along with appropriate gearing and 
linkages, available firom Physik Instruments of Germany (with 
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suppliers in the U.S.) has been found to be suitable for imparting 
these motions. Those skilled in the art will be familiar with the 
design and implementation of the various actuators, high precision 
measurement systems, and the necessary computer control. A few 
5 illustrative embodiments of actuators and measurement systems 

suitable for controlling and determining the position of the imager 14 
will be described in more detail below. 

The lens 12 can be a relatively inexpensive, wide angle lens, 
commercially available from a variety of sources. For example, a 35 

10 millimeter ("mm") camera lens works well for the present invention. 
One suitable imager 14 is an imaging array having a resolution of at 
least 512 x 512 pixels, although imaging arrays of lesser resolution 
will work. Such imaging arrays are available from a variety of 
sources, including Sharp Electronics, Inc. of Japan (with suppliers in 

15 the U.S.). Another contemplated imager 14 is a photosensitive 
integrated circuit having an array of photosensitive elements. One 
advantage in this particular embodiment is that significant processing 
of the signals generated at the photosensitive elements could be 
performed locally within the photosensitive integrated circuit. 

20 In Fig. 2, a multi-camera or "binocular" version of the present 

invention is illustrated. More particularly, an imager apparatus or 
"camera" 22 includes a pair of independent monocular apparatus 10a 
and 10b {see Fig. 1 for corresponding elements) that are operated in 
concert. However, it should be noted that any number n of monocular 

25 apparatus 10 can be used, in concert, for particular applications. For 
example, n=3, 4, 5, or 6 are usefiil for certain applications. 

The apparatus 10a includes a lens 12a, and an imaging array or 
"imager" 14a. As noted by the arrows 16a, 17a, 18a, and 19a, the 


imager 14a is capable of independent in-plane motion with respect to 
the lens 12a for view selection. In certain embodiments, the imager 
10a may also move in the out-of-plane direction for focus control. In 
addition, the imager 14a is preferably synchronized for rectangular 
5 displacements (as described above with reference to Fig. 1) 20a in the 
plane Pa of the imager to provide super-resolution frame integration. 
The various components of the apparatus 10a (e.g. the lens, motion 
actuators, imager, etc.) are as described previously with respect to the 
apparatus 10 of Fig. 1. 

10 Similarly, the apparatus 10b includes a lens 12b, and an 

imaging array or "imager" 14b. As noted by the arrows 16b, 17b, 
18b, and 19b, the imager 14b is capable of independent in-plane 
motion with respect to the lens 12b for view selection. In certain 
embodiments, the apparatus 10b may also move in the out-of-plane 

15 direction for focus control. In addition, the imager 14b is preferably 
synchronized for rectangular displacements (as defined above) 20b in 
the plane Pb of the imager to provide super-resolution frame 
integration. The various components of the apparatus 10b (e.g, the 
lens, motion actuators, imager, etc.) are as described previously with 

20 respect to the apparatus 10 of Fig. 1. 

The apparatus 10a and 10b of camera 22 operate as follows. 
The lateral motions 17a and 19a of apparatus 10a and the lateral 
motions 17b and 19b of apparatus 10b are independent so that the two 
devices can be moved for pan and vergence. However, vertical 
25 motions 16a and 18a of apparatus 10a and 16b and 18b of apparatus 
10b are coupled together either mechanically or through computer 
control for tilt control, as will be appreciated by those skilled in the 
art. Likewise, the rectangular displacements 20a and 20b are 
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synchronized for super-resolution integration. These independent and 
synchronized displacements can be easily accompHshed by computer 
control 21a and 21b (each similar to computer control 21 of apparatus 
10 of Fig. 1 and mutually communicating by a communication link 
5 23) or by a unitary computer control 24. 

In Fig. 3, a greatly enlarged field of coverage 10 of a lens 12 is 
illustrated. Each rectangle 26 represents a potential location of the 
imager 10 in the lens' field of coverage as intended for capture by 35 
mm film. "Additional areas" 28 are not typically used in film 

10 photography, but are accessible to the present invention. In fact, the 
entire area 12 can be sampled by the method and apparatus of this 
invention. For example, a 1/3 inch 4:3 imager provides 5 times 
lateral and 5 times vertical coverage, for an effective field of 3200 by 
2400 (or more) for a nominal 640 by 480 pixel imager 10, selected as 

15 the "standard" image size under NTSC standards. 

As noted previously, in the present invention, an imaging 
device is placed on a three-degree-of- freedom linear motion 
mechanism whose position can be controlled by a computer. 
Relatively broad field coverage is preferably attained through use of a 
20 wide-angle large-coverage ("wide," e.g. 45 degree or larger field of 
view) lens. High resolution is attained by placing the imaging device 
under the wide lens, so the sensor's immediate field of view is 
considerably smaller than that provided by the lens. 

For example, for a 6mm-wide imaging device under a 50mm 
25 lens, the lens' 45 degree field of view is reduced to approximately 45 
X (6 / 35) = 7.7 degrees at the sensor array, making it act as an 
approximately 400 mm lens at the sensor, and increasing pixel 
resolution by a factor of about 6 in each direction. Super-resolution 
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integration techniques can more than double this to a factor of about 
12 in each direction, as will be appreciated by those skilled in the art. 

Computer control of the motion platform upon which the 
sensor array resides enables positioning accuracy in the micron range. 
5 Stepper motors and other precision actuators for accomplishing this 
control are well known to those skilled in the art. Stepper motors 
operating with full step positioning can position a platform at discrete 
positions along an axis to within a few microns. Differential stepper 
control enables increasing the accuracy of this placement by one to 
10 two orders of magnitude. 

For fine determination of absolute imager location, a pair of 
interferometer or sonar-based ranging systems can be provided that 
measure the positions of the sensors in the focal plane to sub-micron 
precision during image capture. Since each imaging cell is about 10 
15 microns across in a current embodiment, this enables sufficient sub- 
pixel precision to keep the range estimates coherent across the 
operational field of view. In the present example, interferometric 
measurement and differential stepper control may provide better than 
3 bits of sub-pixel localization over the range of motion 

20 With reference to Figs. 4 and 5, a method is described for 

utilizing a multi-camera in accordance with the present invention in 
order to adjust the range of available stereoscopic imaging. Fig. 4 
illustrates a multi-camera 22 including a pair of monocular apparatus 
10a and 10b. The corresponding elements of apparatus 10a and 10b 

25 are described above with reference to Figs. 1 and 2. In Fig. 4, the 
imager 14a is centered about a focal axis of the lens 12a and the 
imager 14b is centered about a focal axis of the lens 12b. Given this 
arrangement, the bounds (at least along the plane of the paper) of the 


view selection of the imager 14a are indicated by the lines 110 and 
112, and the bounds (at least along the plane of the paper) of the view 
selection of the imager 14b are indicated by the lines 1 14 and 116. 
The hatched region 1 1 8 indicates range of available stereoscopic 
5 imaging available for the arrangement of Fig. 4, 

In Fig. 5, the imager 14b has been repositioned along the X 
axis and, as a result, the range of available stereoscopic imaging 
available for the arrangement of Fig. 5 includes not only the hatched 
region 1 1 8, but now includes the hatched region 120. By making 
10 similar adjustments along the Y axis, the range of available 
stereoscopic imaging can be varied as desired. 

An important design consideration for the present invention is 
the realization that the alignment of multiple imagers for range 
computation is important (recall the discussion of parallel epipolar 

15 geometry in the background). However, it is contemplated that the 
multiple imagers may be built as independent components and 
arrayed as desired for particular ranging situations. Accordingly, the 
present invention teaches an alignment mechanism operable to align 
the multiple imagers in order to substantially meet the desired parallel 

20 epipolar geometry. One preferred alignment mechanism that will 
enable rapid alignment includes a laser diode system directed parallel 
to the imaging surface of and exiting a first imager housing the laser 
diode. When the imagers are properly aligned, the beam enters 
another imager, is reflected on to the next imager, and so on, such that 

25 the beam retums in a single plane of travel to the originating source. 
In this way, visual alignment of multiple imagers can be obtained to 
the sub-millimeter level with simple integrating signal processing and 


optimization at the detector elements, as will be appreciated by those 
skilled in the art. 

With reference to Figs. 6 and 7 (top view and side view, 
respectively), an alignment mechanism 150 for aligning two moving 
5 imagers in accordance with one embodiment of the present invention 
will now be described. The alignment mechanism 150 is jointly 
housed within moving imagers 152 and 154 and includes a laser 
diode 156, mirrors 158 and 160, and a photosensor 162. The laser 
diode 156, the mirrors 158 and 160, and the photosensor 162 are 

10 arranged such that, when the moving imagers 152 and 154 are 

properly aligned, a beam of light 164 generated at the laser diode 156 
is reflected within the moving imager 152 and directed back to the 
photosensor 162. Thus proper alignment can be achieved by 
adjusting the moving imagers 152 and 154 until the light beam 164 is 

15 detected at the photosensor 162. Mechanical position adjusters, such 
as those available from Nev^ort Inc. of Irvine, Califomia, may be 
used to effect the imager alignment. 

Note that the alignment mechanism 150 locates the laser diode 
156, mirrors 158 and 160, and the photosensor 162 within the 
20 imagers, as opposed to locating them extemally. Doing so enables 
better alignment through properly machining the light paths intemal 
to the imagers. As will further be appreciated, the strategy of Figs. 6 
and 7 can be adapted to align a multiplicity of imagers using a single 
laser diode and additional mirrors. 

25 Another design consideration for the present invention is the 

realization that the precise measurement of location of the sensor is 
more important than positioning accuracy. For most view selections, 
there is no need to achieve a particular exact direction of view: an 


approximate direction will be sufficient. In other words, the sensor 
does not need to be at a specific position under the lens and yet, 
wherever the sensor is placed, knowledge of its position is preferred 
to a quite high precision (e.g. a tenth of a micron). As an illustrative 
5 example, the sensor array could be located at an arbitrary position 
with respect to the imaging plane, as long as the scene detail of 
interest is imaged and the system measures the imager's actual 
location with a high degree of precision. The precision of this 
placement determines the accuracy of the depth computed. 

10 Yet another design consideration for the present invention is 

□ the realization that super-resolution computation is facilitated by 

a having high positioning accuracy over a distance of about 1/2 pixel 

3 (see Tagami et al.'s US Patent No. 5,402,171). The altemative is 

quite acceptable (i.e. knowing to high precision where the other 
15 images contributing to the integration are located). But, knowing the 

□ displacements in advance and then using the known displacements 
u during processing (for example, exactly one half pixel displacement 

in each direction) is a preferable solution. 

There are a number of enhancements to the basic methods and 
20 apparatus described above with reference to Figs. 1-7. It should be 
noted that the present invention includes an instantaneous wide-angle 
viewing capability. Due to this wide-angle viewing capability, the 
requirement of moving the imager over the whole field of view while 
integrating to obtain a full view is eliminated. Instead, preferably 
25 through use of a small positive-curvature converter lens between the 
lens and the imager, the fiiU field of view is simultaneously projected 
onto the centrally-located sensor. 


Another enhancement to the invention is the inclusion of a 
computer-controlled lens focusing mechanism, in addition to the 
back-plane focusing means provided by the platform out-of-plane 
motion, and this would preferably use stereo-computed range to 
5 determine focusing distance. Another enhancement is a computer- 
controlled zoom mechanism that will allow adaptive selection of 
effective field of view over a wide range of values. The 
implementation of computer-controlled lens focusing and zoom 
mechanisms are well known to those skilled in the art. 

10 Thus one particular embodiment of the present invention 

a includes a three-dimensional-adjusting imaging camera that has three 

\j motion platforms coupled for three-dimensional movement, a 

I pj controller for moving the platforms separately or in concert, a 

measurement system for determining the location of the platforms in 

15 an extemal frame of reference, an imaging device positioned for 
Q three-dimensional displacements on the coupled platform, and a lens 

u whose focal surface coincides with the imaging surface of the 

% imaging device under three-dimensional movement of the motion 

platform and which has a field of coverage considerably larger than 

20 the optically sensitive area of the imaging device. 

A three-dimensional-adjusting imaging camera 200 in 
accordance with such an embodiment will now be described with 
reference to Figs. 8 and 9. The imaging camera 200 includes a pair of 
positionable imaging apparatus 202a and 202b that are operated in 
25 concert. The imaging apparatus 202a includes motors 204a, 206a, 
and 208a, linear slides 210a, 212a, and 214a, an imager 216a, and 
three platforms 218a, 220a, and 221a, 


The motor 208a drives the platform 220a along the linear slide 
214a, the motor 204a drives the platform 218a along the linear slide 
212a, and the motor 206a drives the platform 221a along the linear 
sHde 2 10a. The imager 2 1 6a is mounted upon a surface of the 
5 platform 22 la. Motion along the linear sUde 2 14a is perpendicular to 
motion along the linear slide 212a, but motion along both linear slides 
212a and 214a is in-plane motion. Motion along the linear sHde 210a 
is out-of-plane motion. Hence, in-plane positioning is effectuated by 
actuating the motors 204a and 208a, while out-of-plane motion is 
10 effectuated by actuating the motor 206a. Operation of the 

positionable imaging apparatus 202b is similar and should be self- 
evident. 

The camera can be used for the automated tracking of objects 
in the field of view and the automated scanning of a field of view. In 

15 addition, the use of such a camera for integrating images from 
different positions can be used to develop higher resolution 
composite images of the field of view, and for the automated range 
computation to features and elements in the field of view. A lower- 
resolution version of the latter is preferably accompHshed with a 

20 converting lens positioned between a primary lens and imaging 

device surface such that a large portion of the field of view of the lens 
is projected onto the optically sensitive area of the centrally-located 
imager. 

As noted, the present invention may further include a computer 
25 controlled focusing mechanism aside from the back-plane focusing 
means. In addition, the camera of the present invention may further 
include a computer-controlled focal-length adjusting mechanism. 
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A further advantage accruing from moving the imagers under 
computational control is that shuttering and acquisition can be 
synchronized to ensure that images are free of motion blur - either 
from camera motion or from observed object motion. 


applications in that any application of computational video 
processing, including observation, tracking, and stereo ranging, 
would benefit from many elements of the present camera design. 
These benefits include simplified stereo matching, the retention of a 
10 single frame of reference, high resolution and wide field of view with 
inexpensive off-the-shelf lenses, field of view selection without 
camera or lens movement, and super-resolution. 

An alternative to the above-described imaging devices is an 
imaging device which utilizes a spherical lens. The human eye is the 

15 principal capture-device model for theoretical aspects of computer 
vision research. Unfortunately, imager fabrication constraints have 
prevented development of imaging devices that can exploit the 
geometry of this natural "device." Previously, researchers have 
typically only had access to planar focal-surface sensors such as those 

20 described above with reference to Figures 1-9. 

In Fig. 10, a camera 30 includes a spherical lens 31 centered on 
an arbitrary spherical coordinate system and having a lens focal 
sphere 32. The camera 30 further includes an elevation control axis 
34 and an azimuth control axis 36. These axes may be implemented 
25 with curved control paths. A positioner 38 (shown here broken away 
for clarity) is attached to the axes 34 and 36 and is used to hold an 
imager 40 at the intersection of the axes. The actual position of 
positioner 38 is shown at 38'. A spherical-faced fiber-optic faceplate 
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The present invention has a great number of valuable 
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42 is attached over the imager 40 to define the spherical image 
formation surface of the camera. An elevation controller 44 is 
coupled to the elevation control axis 34, and an azimuth controller 46 
is coupled to the azimuth control axis 36. The elevation controller 44 
5 and azimuth controller 46 are preferably computer controlled by a 
computer controller 48. 

A curved focal surface, such as the retina and the spherical 
surface of the faceplate 42 described above, has a number of 
advantages for computational image processing, including: 

10 1) excellent optical resolution attainable when the focal surface 
needn't be made planar; 2) the simplicity of rotational over linear 
displacements; 3) a greatly increased undistorted field of view; 4) 
exponentially less lens illuminance fall-off with eccentricity; 5) 
greater effective pan and tilt velocities; and 6) opportunity to study 

15 (via simulation) aspects of human psychophysical behavior. 

With the present invention, the capture of spherical focal-plane 
imagery through, for example, the use of the spherical-faced fiber- 
optic faceplate 42 provides these and other benefits. The present 
invention provides a mechanism for exploiting this in a spherical- 

20 imaging camera that shares many of the design considerations of the 
above-described linear moving-imager camera. The present design 
uses a high resolution imaging device (IK by IK elements with 12 
bits of resolution at each pixel), mounted behind a fiber-optic 
faceplate that transfers the focal-plane light pattem to the sensor 

25 elements. 

Rotational mechanisms allow azimuth and elevation controls 
on the imager-faceplate's position on a sphere, with mechanisms 
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similar to those used in our linear-motion device described above for 
measuring this position to sub-pixel precision. 

A suitable lens for this embodiment of the present invention is 
a ball lens, which is described in J. Baker 's article entitled 
5 "Spherically Symmetrical Lenses and Associated Equipment for Wide 
Angle Aerial Photography," found in § 16.1 of the Nov. 30, 1949, 

Report of united States Office of scientific Research and 
Development, National defense research Committee, 

which is incorporated herein by reference. Attachment of a fiber- 
10 optic faceplate to such a lens for astronomic study is described in 
Park et al.'s article entitled "Realtime Tracking System for the Wide- 
Field-of- View Telescope Project," found in SPIE VOL. 1111, 
ACQUISITION, TRACKING, AND POINTING III, 196-203 (1989) and 
Lewis et al.'s article entitled "WFOV Star Tracker Camera," found in 
1 5 SPIE VOL 1478, SENSORS AND SENSOR SYSTEMS FOR GUIDANCE 
AND NAVIGATION, 2-12 (1991), both of which are incorporated 
herein by reference. 

A lens similar in design character, although with a narrower 
field of view, is manufactured by the Optical Corporation of America 

20 under the name "Wide Field-of-View Star Tracker." The curvature of 
the faceplate is selected by its radius from the center of rotation. 
Mapping of the resulting image geometry for binocular and motion 
disparity computation is straightforward, as will be appreciated by 
those skilled in the art. A smaller, lower resolution, and mechanically 

25 simpler version of this stereo eyeball camera may be attained by 
placement of a single imaging surface with fiber-optic faceplate 
behind each lens of the two-lens camera. As those skilled in the art 
will appreciate, rotation of the lenses and imagers in this eyeball 


system will not affect the frame of reference, as this imager's lenses 
have no single optical axis, and the interocular baseline remains 
fixed. 

As well as being used for range computation, the spherical- 
5 imaging embodiment of the present invention can be advantageously 
used to study the human eye, examining elements of its motion, 
stereo, and even monocular processing: tracking, saccades, micro 
saccades, foveation, and other such aspects. 

Another aspect of the present invention teaches a method for 
10 synchronization of the moving imager with image capture in order to 
^ J decrease blur resulting from the imager's movement. 

iji The concept of a moving image-capture device presents a 

problem in that moving while capturing induces blur. If capture of 
I^J! blurred images is to be avoided, control of the moving imager's 

15 motions must be synchronized with the image capture process. If this 
i-^ synchronization can be attained, it presents the possibility of 

fy minimizing blur in two forms. First, if all platform motion can be 

i'^ performed in the interval between image capture and the start of the 

next integration period, then no platform-induced motion blur will be 
20 observed — the imagers will be stationary while integrating. Second, 
if a moving object is to be imaged, then movements of the platform 
tailored to the motion of the object will enable higher resolution ~ 
minimal blur - imaging of it and anything else moving with its 
velocity. The preferred embodiment of the moving-imager camera 
25 synchronizes image integration and capture to minimize these capture 
artifacts. 

As will be appreciated by those skilled in the art, video imagers 
generally operate at fixed scan frequencies. A usual mode of capture 


involves having the imager acquire light energy for a period of time 
(the integration time), and then to stop the integration process and 
transmit out the acquired information. The time between the stop of 
capture (the end of the integration) and the start of the next 
integration period presents an opportunity for platform motion. This 
time may be from six to tens of milliseconds, depending on the length 
of the next integration period. In certain embodiments of this 
invention, these timing signals are available to the moving imager 
controller, and platform motion is effected in the period between the 
termination of one integration period and the beginning of the next. 

In another embodiment of this invention, the control signals to 
begin and end image integration are controlled by the moving imager 
computer such that integration is begun only after the intended 
motion is complete. Note that in the latter embodiment a fixed or 
standard image frame rate may not be attained but, for computational 
purposes as opposed to videography purposes, fixed or standard 
frame rates are not essential. Here, the quality and character of the 
acquired imagery are the important factors and, where a clear image is 
desired, the control method described will enable this capture. 

The information used to control use and generation of these 
synchronization timing signals may come from measurement devices 
such as the interferometer or sonar-based systems described earlier, or 
from precalibration experiments in which the velocity and timing 
characteristics of the platforms are measured and tabulated for 
subsequent use. 

While this invention has been described in terms of several 
preferred embodiments, there are alterations, permutations, and 
equivalents which fall within the scope of this invention. As will be 
appreciated by the above descriptions and the drawings, the imager 


control methods and apparatus of the present invention provide a 
stable frame of reference for monocular and multiple image computer 
analysis for high-precision tracking, scene scanning, and range 
computation over a wide field of view and with detail of interest at a 
wide variety of distances. However, it should be noted that those 
skilled in the art will realize that there are alterations, permutations, 
and equivalents as fall within the true spirit and scope of the present 
invention. 


